Papers with Visual Grounding
Multimodal Incremental Transformer with Visual Grounding for Visual Dialogue Generation (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing studies focus on implicit exploration of multimodal coreference but neglect the importance of locating the objects explicitly in the visual content, which is associated with textual entities. |
| Approach: | They propose a multimodal incremental transformer with visual grounding which aims to explicitly locate related objects in the image guided by textual entities. |
| Outcome: | The proposed model achieves comparable performance on the VisDial v0.9 and v1.0 datasets. |
LLMs as Bridges: Reformulating Grounded Multimodal Named Entity Recognition (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for Grounded Multimodal Named Entity Recognition (GMNER) lack a strong correlation between image-text pairs and is ungroundable. |
| Approach: | They propose a framework that reformulates GMNER into a joint MNER-VE-VG task by leveraging large language models as a connecting bridge. |
| Outcome: | The proposed framework outperforms state-of-the-art methods on the existing GMNER dataset and achieves absolute leads of 10.65%, 6.21%, and 8.83% in all three subtasks. |
Measuring Faithful and Plausible Visual Grounding in VQA (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Lack of visual grounding (VG) in VQA systems can manifest in over-reliance on irrelevant image parts or a disregard for the visual modality entirely. |
| Approach: | They propose a new metric that measures if a model identifies question-relevant objects in the scene and relies on the information contained in the relevant objects when producing its answer. |
| Outcome: | The proposed metric measures if a model identifies question-relevant objects in the scene and relies on the information contained in the relevant objects when producing its answer. |
Uncovering the Full Potential of Visual Grounding Methods in VQA (2024.acl-long)
Copied to clipboard
| Challenge: | Visual Grounding (VG) methods in VQA aim to strengthen a model's reliance on question-relevant visual information. |
| Approach: | They propose to strengthen a model's reliance on question-relevant visual information by using a visual grounding method that is based on a question-related visual input. |
| Outcome: | The proposed methods can be much more effective when evaluation conditions are corrected. |
ViGiL3D: A Linguistically Diverse Dataset for 3D Visual Grounding (2025.acl-long)
Copied to clipboard
| Challenge: | 3D visual grounding models localize entities in a scene referred to by natural language text . recent studies focused on LLM-based scaling of 3DVG datasets, but these do not capture the full range of potential prompts which could be specified in the English language. |
| Approach: | They propose a framework for linguistically analyzing 3DVG prompts and introduce a diagnostic dataset for evaluating 3D visual grounding methods against a diverse set of language patterns. |
| Outcome: | The proposed framework scales up and tests against a representative set of prompts in the english language. |